Currently it's really easy to reproduce a memory "leak" with xml-conduit where a part of the Pipe becomes a CAF and gets evaluated arbitrarily deeply as the parser runs, while simultaneously being retained at the root because it's a CAF. Exactly how this works is explained at https://www.well-typed.com/blog/2016/09/sharing-conduit/ .

The shotgun approach to this (mentioned in the blog-post) is to disable full laziness and I see a lot of people doing this (c.f. #142 and snoyberg/conduit#370)

However I decided to look deeper into it: the more nuanced advice that the article gives is that conduit pieces that do not depend on any parameters are specifically at risk of becoming CAFs. From profiling with -hd and reading the Core I found out that this is exactly what happens to addBeginEnd.

I wasn't able to reliably prevent it from becoming a CAF (the article advises separating it into a module compiled separately under -fno-full-laziness), however I was able to increase sharing so that the CAF loops back on itself and occupies constant space. This also fixes the memory leak and is also given as an advice in the article.

The culprit seems to be replacing a fixpoint on ConduitT with a fixpoint on Pipe, which is how awaitForever is implemented.

I don't claim to understand this issue fully, but this fixes the leaks in my use-cases. I believe this also addresses #136